Image processing apparatus and image processing method, and program

ABSTRACT

An apparatus for and a method of executing noise reduction processing and defect compensation processing on an image in an RGBW arrangement are provided. In pixel value compensation processing of a color pixel that makes up image data in the RGBW arrangement that has each color pixel of R, G, and B and a white (W) pixel, the W pixel is interpolated at a position of an attention pixel that is a compensation target, and at a position of a reference pixel which has the same color as an attention pixel within a reference area, smoothing weight is calculated based on each pixel value of the interpolation W pixel, and thus a compensation pixel value of the attention pixel is calculated by executing smoothing processing to which the calculated smoothing weight is applied. Moreover, by applying the W pixel in the neighborhood of the color pixel, it is determined whether or not the color pixel is in a texture area, and only if the color pixel is not in the texture, defect compensation processing is executed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is the National Stage of International Application No.PCT/JP2012/080872, filed in the Japanese Patent Office as a ReceivingOffice on Nov. 29, 2012, which claims priority to Japanese PatentApplication Number 2012-011716, filed in the Japanese Patent Office onJan. 24, 2012, each of which is hereby incorporated by reference in itsentirety.

TECHNICAL FIELD

The present disclosure relates to an image processing apparatus and animage processing method, and a program, and relates particularly to animage processing apparatus for and an image processing method of, and aprogram for performing compensation of pixels including noise ordefective pixels that make up an image.

BACKGROUND ART

For a filter that is used in an imaging element in an imaging apparatus,such as a digital camera, for example, a Bayer arrangement and the likein which R, G, and B colors are arranged is used in most cases, but afilter has been proposed that has an RGBW arrangement includingall-wavelength transmission type white (W) pixels, which includes allwavelength areas of the R, G, and B in addition to the R, G, and Bcolors.

However, problems with reduction processing of noise that is included inan image that is captured using the RGBW arrangement filter and that hasthe white (W) pixels are as follows. Color pixels of the R, G, B and thelike are lower in sensitivity and are greater in an amount of noise thanthe white (W) pixels, and furthermore, the number of reference pixelsthat are usable to calculate a compensation pixel value of an attentionpixel that is a noise reduction target, that is, the number of samplesof the reference pixels which have the same color as a compensationtarget pixel is small. As a result, there is a problem in that eventhough the noise reduction (NR) processing that refers to the pixelswhich have the same color is executed, a sufficient noise reductioneffect cannot be obtained.

Furthermore, if an object is to address a model, like a light shotnoise, in which noise changes according to optical strength, this causesa problem in that smoothing strength varies widely due to noise stayingon the pixel itself and a signal level is lowered.

Furthermore, because color-pixel sampling positions are scattered indefect compensation in a white (W) arrangement, it is difficult todistinguish between a defect and a texture, and there is a problem inthat an effect of sufficient compensation cannot be obtained.

Moreover, as the related art relating to the processing that reducesnoise in the image is disclosed, there are, for example, PTL 1 (JapaneseUnexamined Patent Application Publication No. 2003-224861), PTL 2(Japanese Unexamined Patent Application Publication No. 2011-76186, andthe like.

PTL 1 (Japanese Unexamined Patent Application Publication No.2003-224861) discloses a configuration in which noise is reduced byreducing a frequency component of a color difference signal (C) of eacharea according to a frequency component strength of a luminance signal(Y). However, because this technique performs reduction of noise in theC signal, based on the Y signal even at a point where there is norelationship between the luminance signal (Y) and the color differencesignal (C), there is a concern in that the color difference signal willbe lost at a point where a luminance change rarely occurs, such as acolor texture.

In PTL 2 (Japanese Unexamined Patent Application Publication No.2011-76186), there is disclosed a technique that performs determinationof a texture direction using a W pixel, and based on the result of thedetermination, performs the defect compensation. PTL 2 discloses thetechnique of performing defect compensation on the W pixel, but not amethod of performing compensation on color pixels other than W.Furthermore, there is a problem in that an arithmetic operation cost isincreased because of a variety of direction determination processingsubject to two-dimensional processing that refers to pixels in upward,downward, leftward and rightward directions on a two-dimensional plane.

CITATION LIST Patent Literature

PTL 1: Japanese Unexamined Patent Application Publication No.2003-224861

PTL 2: Japanese Unexamined Patent Application Publication No. 2011-76186

SUMMARY OF INVENTION Technical Problem

In view of the problems described above, an object of the presentdisclosures is to provide an image processing apparatus for and an imageprocessing method of, and a program for performing reduction in noisethat is included in an image that is captured through a filter equippedwith an all-wavelength transmission type white (W) pixel or performingdefect compensation.

Solution to Problem

The first aspect of the present disclosure provides an image processingapparatus including: a signal processing unit that executes pixel valuecompensation, in which the signal processing unit inputs image data inan RGBW arrangement that has each color pixel of R, G, and B and a white(W) pixel that passes through almost all wavelength light of eachwavelength of the R, G, and B, and in which the signal processing unitinterpolates the W pixel at a position of an attention pixel that is acompensation target and at a position of a reference pixel which has thesame color as the attention pixel within a reference area, in a pixelvalue compensation processing of a color pixel, calculates smoothingweight based on each pixel value of the interpolation W pixel, and thuscalculates a compensation pixel value of the attention pixel byexecuting smoothing processing to which the calculated smoothing weightis applied.

In an embodiment of the image processing apparatus according to thedisclosure, the signal processing unit may determine whether or not oneor more saturation pixel values are present in the pixel values of theinterpolation W pixel, and if the saturation pixel value is not presentin the pixel values of the interpolation W pixel, the signal processingunit may calculate a compensation pixel value of the attention pixel byexecuting the smoothing processing to which the smoothing weight,calculated based on the each pixel value of the interpolation W pixel,is applied, and if the saturation pixel value is present in the pixelvalues of the interpolation W pixel, the signal processing unit maycalculate the compensation pixel value of the attention pixel byexecuting the smoothing processing to which the smoothing weight,calculated based on each pixel value of the attention pixel that is thecompensation target, and of the reference pixel which has the same coloras the attention pixel within the reference area, is applied withoutapplying the interpolation W pixel.

In an embodiment of the image processing apparatus according to thedisclosure, the signal processing unit may execute processing thatinterpolates the W pixel at the position of the reference pixel whichhas the same color as the attention pixel present in the reference areathat is a two-dimensional area with the reference area as thetwo-dimensional area.

In an embodiment of the image processing apparatus according to thedisclosure, the signal processing unit may execute processing thatinterpolates the W pixel at the position of the reference pixel whichhas the same color as the attention pixel present in the reference areathat is a one-dimensional area with the reference area as theone-dimensional area.

In an embodiment of the image processing apparatus according to thedisclosure, the signal processing unit may execute the pixel valuecompensation as noise reduction (NR) processing that reduces noise thatis included in the attention pixel.

In an embodiment of the image processing apparatus according to thedisclosure, the signal processing unit may determine a likelihood of adefect, that is, determines whether or not there is a likelihood thatthe color pixel will be a defective pixel, in which the signalprocessing unit may execute texture detection processing that determineswhether or not the color pixel is in a texture area, by applying the Wpixel in the neighborhood of the color pixel that is determined ashaving the likelihood of the defect, and in the texture detectionprocessing, if it is determined that the color pixel is in the texturearea, the signal processing unit may not execute defect compensationprocessing, and in the texture detection processing, if it is determinedthat the color pixel is not in the texture area, the signal processingunit may execute the defect compensation processing.

In an embodiment of the image processing apparatus according to thedisclosure, in the texture detection processing, the signal processingunit may determine whether or not the color pixel is in the texturearea, by applying a difference in the pixel value between the W pixelthat is closest to the color pixel that is determined as having thelikelihood of the defect, and the W pixel outside of the closest Wpixel.

The second aspect of the present disclosure provides an image processingmethod of executing pixel value compensation in an image processingdevice, in which a signal processing unit of the image processing deviceperforms: inputting image data in an RGBW arrangement that has eachcolor pixel of R, G, and B and a white (W) pixel that passes throughalmost all wavelength light of each wavelength of the R, G, and B;interpolating the W pixel at a position of an attention pixel that is acompensation target, and at a position of a reference pixel which hasthe same color as the attention pixel within a reference area, in apixel value compensation processing of a color pixel; calculatingsmoothing weight based on each pixel value of the interpolation W pixel;and calculating a compensation pixel value of the attention pixel byexecuting smoothing processing to which the calculated smoothing weightis applied.

The third aspect of the present disclosure provides a program forexecuting pixel value compensation in an image processing apparatus,which causes a signal processing unit of the image processing apparatusto perform: inputting image data in an RGBW arrangement that has eachcolor pixel of R, G, and B and a white (W) pixel that passes throughalmost all wavelength light of each wavelength of the R, G, and B;interpolating the W pixel at a position of an attention pixel that is acompensation target, and at a position of a reference pixel which hasthe same color as the attention pixel within a reference area, in apixel value compensation processing of a color pixel; calculatingsmoothing weight based on each pixel value of the interpolation W pixel;and calculating a compensation pixel value of the attention pixel byexecuting smoothing processing to which the calculated smoothing weightis applied.

Moreover, it is possible that the program according to the presentdisclosure, for example, is provided using a storage medium or acommunication medium that is provided in a computer-readable format toan information processing apparatus or a computer•system that is capableof executing a variety of programs•codes. By providing such a program inthe computer-readable format, the processing according to the program onthe information processing apparatus or the computer•system is realized.

Other objects, features, and advantages according to the presentdisclosure are apparent from examples according to the presentdisclosure, which are described below, or from a more detaileddescription that is based on that attached drawings. Moreover, thesystem in the present specification is configured to be a logicalcombination of multiple apparatuses, and the apparatuses in eachconfiguration are not limited to being within the same housing.

Advantageous Effects of Invention

With a configuration of an example according to the present disclosure,an apparatus for and a method of executing noise reduction processingand defect compensation processing on an image in an RGBW arrangementare realized.

Specifically, in pixel value compensation processing of a color pixelthat makes up image data in the RGBW arrangement that has each colorpixel of R, G, and B and a white (W) pixel, the W pixel is interpolatedat a position of an attention pixel that is a compensation target, andat a position of a reference pixel which has the same color as theattention pixel within a reference area, smoothing weight is calculatedbased on each pixel value of the interpolation W pixel, and thus acompensation pixel value of the attention pixel is calculated byexecuting smoothing processing to which the calculated smoothing weightis applied. Moreover, by applying the W pixel in the neighborhood of thecolor pixel, it is determined whether or not the color pixel is in atexture area, and only if the color pixel is not in the texture, defectcompensation processing is executed.

With such processing, the apparatus for and the method of executing thenoise reduction processing and the defect compensation processing on theimage in the RGBW arrangement are realized.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for describing an example of an RGBW arrangement.

FIG. 2 is a diagram for describing a decrease in a density of colorpixels in the RGBW arrangement.

FIG. 3 is a diagram for describing a specific example of noise reductionprocessing that is executed in an image processing apparatus accordingto the present disclosure.

FIG. 4 is a diagram for describing a specific example of a smoothingfunction that is applied to the noise reduction processing that isexecuted in the image processing apparatus according to the presentdisclosure.

FIG. 5 is a diagram for describing a specific example of the noisereduction processing that is executed in the image processing apparatusaccording to the present disclosure.

FIG. 6 is a diagram for describing a merit of obtaining smoothingstrength using a W pixel.

FIG. 7 is a diagram for describing a countermeasure against the W pixelbeing saturated in the noise reduction processing that is executed inthe image processing apparatus according to the present disclosure.

FIG. 8 is a diagram for describing an example of processing that has tobe executed if assigning of and compensation of an interpolation W pixelare executed using one-dimensional (1D) data line by line.

FIG. 9 is a diagram illustrating a flow chart for describing a detailedsequence for the noise reduction processing that is executed in theimage processing apparatus according to the present disclosure.

FIG. 10 is a diagram for describing an example of an RGBW arrangement.

FIG. 11 is a diagram for describing an outline of defect compensationprocessing.

FIG. 12 is a diagram for describing an example of general detectionprocessing of a pixel defect.

FIG. 13 is a diagram for describing an example of the general detectionprocessing of the pixel defect.

FIG. 14 is a diagram for describing one example of processing thatdetects a texture from an image, which is executed in the imageprocessing apparatus according to the present disclosure.

FIG. 15 is a diagram illustrating a flow chart for describing aprocessing sequence for the defect compensation processing that isexecuted in the image processing apparatus according to the presentinvention.

FIG. 16 is a diagram for describing a configuration of and processing bythe image processing apparatus according to the present disclosure.

FIG. 17 is a diagram for describing the configuration of and theprocessing by the image processing apparatus according to the presentdisclosure.

FIG. 18 is a diagram for describing the configuration of and theprocessing by the image processing apparatus according to the presentdisclosure.

DESCRIPTION OF EMBODIMENTS

An image processing apparatus, an image processing method, and a programaccording to the present disclosure are described in detail belowreferring to the drawings. Moreover, descriptions are provided belowunder the following headings.

1. Pixel Arrangement Including All-Wavelength Transmission Type W(White) Pixels

2. Example of Processing that Calculates a Compensation Pixel Value of aColor Pixel by Interpolating a W Pixel at a Pixel Position of the ColorPixel that is a Compensation Target and Applying the Interpolation WPixel

3. Example of Processing that Performs Defect Compensation according toa Result of Texture Detection to which the W pixel is applied inCompensation Processing of the Color Pixel that is a Defect CompensationTarget

4. Configuration example of an Imaging Processing Apparatus

4-1. Configuration Example 1 of the Image Processing Apparatus

4-2. Configuration Example 2 of the Image Processing Apparatus

4-3. Configuration Example 3 of the Image Processing Apparatus

5. Conclusions of Configurations of the Present Disclosure

[1. Pixel Arrangement Including all-Wavelength Transmission Type W(White) Pixels]

First, an example of a pixel arrangement that includes all-wavelengthtransmission type W (white) pixels, that is, an example of anarrangement of pixels of the imaging element capturing an image that isa target for noise reduction processing or defect compensationprocessing in the image processing apparatus according to the presentdisclosure is described.

As described above, in an imaging apparatus such as a digital camera, aBayer arrangement in which R, G, and B colors are arranged as filtersused in the image element is widely known, but recently, a filter hasbeen proposed that has an RGBW arrangement that includes all-wavelengthtransmission type W (white) pixels that allow light of the R, G, and Bcolors in almost all wavelength ranges of the R, G, and B colors to passthrough.

A specific example of the RGBW arrangement is an RGBW arrangement thatis illustrated in FIGS. 1( a) to 1(d). Moreover, FIG. 1( e) illustrates,as a reference example, a Bayer arrangement that is a general RGBarrangement.

As illustrated in FIG. 1, various arrangements, each of which has the Wpixels, have been proposed, but by using the W pixel, a color pixeldensity of the R, G, and B colors is decreased compared to a Bayerarrangement in the related art.

In most cases, as processing that reduces noise included in the image,processing is performed that selects as the reference pixelsneighborhood pixels that have the same color as a noise reduction-targetpixel (attention pixel) and calculates the compensation pixel value ofthe noise reduction pixel using the pixel values of the referencepixels.

However, for example, the density of the R, G, and B pixels in the RGBWarrangements that are illustrated in FIGS. 1( a) to 1(d) is decreasedcompared to the Bayer arrangement that is illustrated in FIG. 1( e).Therefore, the reference pixels that are usable in a case where thecompensation processing is performed for the purpose of noise reductionin each of the R, G, and B pixels, that is, the neighborhood pixels thathave the same color as the compensation target pixel and that areavailable for reference are decreased in number. That is, the number ofthe color pixels that have the same color per unit area is decreased. Asa result, a problem that the pixel values of the reference pixels ofwhich the number is sufficient cannot be used and compensation precisionis decreased occurs.

A specific example is described referring to FIG. 2.

FIG. 2 is a diagram illustrating the 7×7 pixel area in each of the RGBWpixel arrangements illustrated in FIGS. 1( a) and 1(c).

For example, in FIG. 2( a), the central pixel (B) in the 7×7 pixel areais set to be the attention pixel, that is, the noise reduction-targetpixel. The 7×7 pixel area of which the center is the attention pixel isset to be the reference area of which the center is the attention pixel.Processing is executed that calculates the compensation pixel value ofthe attention pixel, that is, a noise reduction pixel value by selectingas the reference pixel the pixel that has the same color as theattention pixel which is included in the reference area and by using thepixel value of the reference value.

However, as illustrated in FIG. 2( a), as few as four B pixels that havethe same color as the B pixel that is a central pixel are included in a7×7 pixel reference area.

Moreover, if the noise reduction processing is performed, for example,processing is performed that calculates the compensation pixel value ofthe attention pixel by assigning weight that depends on the differencein the pixel value between the attention pixel which is the compensationtarget pixel and the reference pixel and by executing addition forweighting and the like. That is, compensation processing that changesthe pixel value of the attention pixel, so-called smoothing processingis performed in such a manner as to smooth the pixel values of theattention pixel and the reference pixel in the vicinity of the attentionpixel that has the same color as the attention pixel. However, when thenumber of the reference pixels is small, for example, when even oneerror pixel is included in in the compensation target pixels or thereference pixels, or in other cases, there is a problem in that a trendof calculating the unnatural compensation pixel value becomes strong.

This is true also for the pixel arrangement that is illustrated in FIG.2( c), or for other arrangements including the RGBW arrangement.Moreover, also in the R and G pixels other than the B pixel, there is aproblem in that the number of the pixels that have the same color perunit area in the RGBW arrangement is smaller than in the RGB pixelarrangement and the compensation precision is decreased as the number ofthe reference pixels applicable for calculating the compensation pixelvalue is decreased.

[2. Example of Processing that Calculates a Compensation Pixel Value ofa Color Pixel by Interpolating a W Pixel in a Pixel Position of theColor Pixel that is a Compensation Target and Applying the InterpolatedW Pixel]

Next, an example of processing that calculates the compensation pixelvalue of the color pixel by interpolating the W pixel in a pixelposition of the color pixel that is a compensation target and applyingthe interpolation W pixel is described as an example of the image noisereduction processing that is executed in the image processing apparatusaccording to the present disclosure.

As described above, for example, in the RGBW arrangements that areillustrated in FIGS. 1( a) to 1(d) or FIGS. 2( a) and 2(c), the pixeldensity of the R, G, and B color pixels is decreased compared to theBayer arrangement that is configured from only the RGB pixels, which isillustrated in FIG. 1( e).

For example, in the reference areas made from 7×7 pixels, of which thecenters are the attention pixels that are illustrated in FIG. 2, thenumber of the pixels that have the same color as the attention pixel inthe center that is the compensation target is small, and thecompensation precision is decreased in the compensation processing thatrefers to the pixels having the same color, of which the number issmall.

In the image processing apparatus according to the present disclosure,the compensation pixel value of the color pixel is calculated byinterpolating the W pixel at the pixel position of the color pixel thatis the compensation target and applying the interpolation W pixel. Anexample of the processing is described below referring to FIG. 3 andsubsequent figures.

A captured image (mosaic image) in FIG. 3(1) is an image that has thesame arrangement as that in FIG. 2( a), and indicates an output image inthe RGBW arrangement, of the imaging element. Each pixel of the outputimage of the imaging element is set to only the pixel value of any oneof R, G, B, and W.

Moreover, such an image is called the mosaic image. Processing thatassigns the pixel values of R, G, and B to each pixel position in themosaic image is called de-mosaic processing, and for example, in adigital camera, the image in which the RGB pixel value is assigned toeach pixel position by executing the de-mosaic processing is stored in amemory and is displayed on a display.

Examples in which pixel-based noise reduction processing is performed onthe mosaic image that is present before the de-mosaic processing andthat is illustrated in FIG. 3(1) are illustrated in FIG. 3.

The examples in FIG. 3 illustrate a case where the compensation pixelvalue of the compensation target pixel is calculated by setting the 7×7pixel reference area of which the center is the pixel that is thecompensation target pixel, that is, a noise reduction target. In theexamples in FIG. 3, a B pixel (B0) that is the center of the 7×7 pixelreference area is set to be the compensation target pixel, and thecompensation pixel value of the B pixel (B0) is calculated.

First, the pixel that has the same color as the compensation targetpixel is selected from the reference area. Four pixels B1, B2, B3, andB4 that are illustrated in FIG. 3(1) are selected as the referencepixels.

For example, calculation of the following compensation pixel value (B0′)is executed if the noise reduction processing in the related art isperformed, that is, if the so-called smoothing processing is performedthat calculates the compensation pixel value of the attention pixel byexecuting the addition for weighting and the like that depends on thedifference in the pixel value between the attention pixel that is thecompensation target pixel and the reference pixel.B0′=p(B0)|B0−B0|+q(B1)|B0−B1|+r(B2)|B0−B2|+s(B3)|B0−B3|+t(B4)|B0−B4|  (Equationa)

In (Equation a) described above, p, q, r, s, and t are weights(smoothing weights) that depend on the difference in the pixel valuebetween the attention pixel and the reference pixel, and for example, asmoothing function that defines a curved line illustrated in FIG. 4 isstipulated in advance and p, q, r, s and t are set according to thesmoothing function.

In this manner, if the noise reduction processing is performed, forexample, the so-called smoothing processing is performed many times thatcalculates the compensation pixel value of the attention pixel byexecuting the addition for weighting and the like that depend on thedifference in the pixel value between the attention pixel that is thecompensation target pixel and the reference pixel. However, if thenumber of the reference pixels is small and the addition for weightingthat depends on the difference in the pixel value between the attentionpixel and the reference pixel, for example, when even one error pixel isincluded, a case occurs in which an effect of the error pixel isincreased and an unnatural compensation pixel value is calculatedwithout performing optimal smoothing.

In the noise reduction processing, that is, pixel value compensationprocessing, which is executed on the attention pixel by the imageprocessing apparatus according to the present disclosure, the W pixelvalue of each position of the attention pixel (B0) that is thecompensation target and the four reference pixels (B1 to B4) isestimated. That is, W pixel interpolation processing is performed oneach pixel position of the attention pixel (B0) that is the compensationtarget and the four attention pixels (B1 to B4). The W pixelinterpolation processing is executable by interpolation processing thatsets the W pixel in the vicinity of an interpolation pixel position tobe the reference pixel.

In the RGBW arrangements that are illustrated in FIG. 3, approximatelyhalf of the 7×7 pixels that are illustrated as the reference area areconfigured from the W pixels, and the pixel density of the W pixels isthe highest. For example, the four W pixels are present, adjacentlyupward, downward, leftward, and rightward, in the vicinity of each ofthe five pixel positions of the attention pixel (B0) and the fourreference pixels (B1 to B4). In each interpolation pixel position, forexample, an average of the pixel values of the four W pixels iscalculated, and thus the W pixel value of the interpolation W pixel isdetermined. Alternatively, an interpolation pixel value may becalculated with the interpolation processing in which an edge directionis considered, for example, with the addition for weighting and the likein which a large weight is assigned to the pixel in a decreasingdirection of a gradient of the pixel value.

Like in an interpolation image illustrated in FIG. 3(2), interpolation Wpixels B4(W0 to W5) are assigned to the pixel positions of the attentionpixel (B0) that is the compensation target and the four reference pixels(B1 to B4), respectively.

The compensation pixel value that results after the reduction in noisein the attention pixel (B0) that is the compensation target iscalculated by applying the W pixel value of the interpolation W pixel.

Specifically, the compensation pixel value is calculated by applyingsmoothing strength that is calculated using the interpolated W pixelvalue as illustrated in FIG. 5.

For example, a compensation pixel value I_(NR)(P) of the attention pixelis calculated according to (Equation 1) illustrated below.

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 1} \right\rbrack & \; \\{{I_{NR}(p)} = \frac{\sum\limits_{q \in \Omega_{p}}{{I(q)} \cdot {\varphi\left( {{{W(q)} - {W(p)}}} \right)}}}{\sum\limits_{q \in \Omega_{p}}{\varphi\left( {{{W(q)} - {W(p)}}} \right)}}} & \left( {{Equation}\mspace{14mu} 1} \right)\end{matrix}$

In Equation described above, parameters indicate the following values.

I_(NR)(p): the compensation pixel value (=compensation pixel value thatresults after the noise reduction processing) of the attention pixel

Ωp: the reference area of which the center is the attention pixel

I(q): the pixel value of the reference pixel that is the pixel which hasthe same color as the attention pixel within the reference area Ωp

W(p): the interpolation W pixel value of a position of the attentionpixel

W(q): the interpolation W pixel value of a position of the referencepixel that has the same color as the attention pixel within thereference area Ωp

φ: the smoothing function

The compensation pixel value of the attention pixel that is the centerof the reference area is calculated according to (Equation 1) describedabove. From (Equation 1) described above, the compensation pixel valueof the attention pixel is calculated by determining the smoothing weightusing the interpolation W pixel value of the position of the referencepixel.

The φ: smoothing function is the same function as described abovereferring to FIG. 4, and is a function that assigns the weight thatdepends on the difference between the attention pixel and the referencepixel. Moreover, in the processing example, the φ: smoothing function isdefined as a function that calculates the weight that depends on thedifference between the W pixel value of the interpolation W pixel thatcorresponds to the position of the attention pixel and the W pixel valueof the interpolation W pixel that corresponds to the position of thereference pixel.

In this manner, for example, the number of the B pixels that is includedin the 7×7 pixel reference area is small, but the number of W pixelsthat are included in the 7×7 pixel reference area is large, and thenoise reduction processing is possible with greater precision by usingthe B pixel, as is, as the reference pixel, thus determining thesmoothing weight using the interpolation W pixel, and thus calculatingthe interpolation pixel value.

Particularly, because a strong relationship between the W pixel and thecolor pixel of each of the R, G, and B is present that results fromspectral characteristics of a color filter, this produces an effectiveresult. Furthermore, because the W pixel is high in sensitivity, noiseis decreased, and it is possible to determine the appropriate smoothingweight that depends on an edge or a texture. Moreover, because there isno relationship between noise staying on the W pixel and noise stayingon the color pixel, there is also an advantage in that a change in asignal level is suppressed without relying on the pixel value on whichsmoothing is performed directly with the smoothing strength, bycalculating the smoothing strength from the W pixel.

Moreover, the processing example that is described referring to FIGS. 3to 5 is a processing example in which the compensation pixel value ofthe B pixel is calculated with the attention pixel as the B pixel, butthe same processing is performed also on the R and G that are colorpixels other than the B pixel.

That is, the compensation pixel value is calculated that reduces thenoise by setting the reference area (for example, the 7×7 pixels) ofwhich the center is any color pixel of the R, G, and B color pixels,which is the compensation target, interpolating the W pixel at thepositions of the attention pixel and the reference pixel, and applying(Equation 1) described above.

Moreover, the reference area is not limited to the 7×7 pixels, and it ispossible to set variously-sized areas other than the 7×7 pixel area.

In the present processing example, the compensation pixel value that isthe noise reduction pixel value of the attention pixel is calculatedaccording to (Equation 1) described above, but in the noise reduction(NR) processing, a merit of obtaining the smoothing strength using the Wpixel is described referring to FIG. 6.

FIG. 6 illustrates subsequent processing examples.

(A) An example in which the signal level changes when the noisereduction (NR) processing is executed if the smoothing strength iscalculated from the color pixel

(B) An example in which the signal level changes when the noisereduction (NR) processing is executed if the smoothing strength iscalculated from the interpolation W pixel

That is, FIG. 6(A) illustrates the processing that, when described withthe pixel value compensation of a B0 pixel being illustrated in FIG. 3as an example, calculates the compensation pixel value (B0′) accordingto (Equation a) described above, that is, (Equation b) described below.B0′=p(B0)|B0=B0|+q(B1)|B0−B1|+r(B2)|B0−B2|+s(B3)|B0−B3|+t(B4)|B0−B4|  (Equation b)

On the other hand, FIG. 6(B) is equivalent to processing that calculatesthe compensation pixel value according to (Equation 1) described above.

As illustrated in FIG. 6(A), if the smoothing strength is obtained fromthe color image itself, when positive noise stays on the central pixelthat is the compensation target, strong compensation (NR) is applied andthe signal is greatly decreased. On the other hand, if negative noisestays, weak compensation (NR) is applied, and an increase in the signalis small. In total, the signal level is lowered.

On the other hand, as illustrated in FIG. 6(B), by obtaining thesmoothing strength from the interpolation W pixel, it is possible toapply the compensation (NR) to almost the same extent not only in a casewhere the positive noise stays on the central pixel that is thecompensation object but also in a case where the negative noise isapplied, and In total, the compensation that decreases the change in thesignal level is possible.

In the configuration according to the present disclosure, a dot to whichthe compensation (NR) is applied strongly and a dot to which thecompensation (NR) is applied weakly are set to be at random withoutrelying on the level of noise staying on the central pixel, by using thefact that there is no relationship between the noise staying on the RGBand the noise staying on the W and by calculating the smoothing strengthfrom the interpolated W pixel. Accordingly, it is possible that anaverage level of the signal that results after the compensation (NR)approaches a central value.

In this manner, in the configuration in which the smoothing weight isassigned by interpolating the W pixel at the position of each colorpixel of the R, G, and B and using the interpolation W pixel value, oneproblem is when the W pixel is saturated. There is a high likelihoodthat the pixel that has the saturation pixel value will not reflect thecorrect pixel value, and it is preferable that the compensation in whichthe saturation pixel is set to be the reference pixel not be performed.

A countermeasure against the W pixel being saturated is described belowreferring to FIG. 7.

The W pixel is interpolated at the positions of the attention pixel andthe reference pixel by setting the reference area (for example, the 7×7pixels) of which the center is any color pixel of the R, G, and B colorpixels, which is the compensation target.

The processing described so far is the same as that described abovereferring to FIGS. 3 to 5.

As illustrated in FIG. 7, the interpolation pixel W0 is assigned to theposition of the attention pixel that is the compensation target in thecenter of the reference area, and the four interpolation W pixels W1 toW4 are assigned to the 7×7 pixel reference area in the vicinity of theattention pixel.

Next, a saturation processing that determines whether or not any one ofthe interpolation W pixels is saturated is performed.

If any one of the interpolation W pixels is a maximum pixel value, it isdetermined that such an interpolation W pixel is saturated.

If none of the interpolation W pixels W0 to W4 are the saturationpixels, as illustrated in FIG. 7( a), according to (Equation 1)described above, compensation (NR) processing is executed by applyingthe smoothing weight that is calculated by applying the interpolation Wpixel without applying the interpolation W pixels, and thus bycalculating the compensation pixel value of the attention pixel.

On the other hand, if even one saturation pixel is included in theinterpolation W pixels W0 to W4, as illustrated in FIG. 7( b), accordingto (Equation 2) described below, the compensation (NR) processing isexecuted by applying the smoothing weight based on the differencebetween the attention pixel and the reference pixel that has the samecolor as the attention pixel without applying the interpolation W pixel,and thus by calculating the interpolation value of the attention pixel.

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 2} \right\rbrack & \; \\{{I_{NR}(p)} = \frac{\sum\limits_{q \in \Omega_{p}}{{I(q)} \cdot {\varphi\left( {{{I(q)} - {I(p)}}} \right)}}}{\sum\limits_{q \in \Omega_{p}}{\varphi\left( {{{I(q)} - {I(p)}}} \right)}}} & \left( {{Equation}\mspace{14mu} 2} \right)\end{matrix}$

In Equation described above, parameters indicate the following values.

I_(NR)(p): the compensation pixel value (=compensation pixel value thatresults after the noise reduction processing) of the attention pixel

Ωp: the reference area of which the center is the attention pixel

I(q): the pixel value of the reference pixel that is the pixel which hasthe same color as the attention pixel within the reference area Ωp

φ: the smoothing function

In this manner, (a) if the pixel value of the interpolation W pixel isnot saturated, the compensation (NR) processing is executed by applyingthe smoothing weight that is calculated by applying the interpolation Wpixel, and thus by calculating the interpolation pixel value of theattention pixel.

(b) If the pixel value of the interpolation W pixel is saturated, thecompensation (NR) processing is executed by applying the smoothingweight that is calculated by applying the original color pixel, not theinterpolation W pixel, and thus by calculating the interpolation pixelvalue of the attention pixel.

In this manner, the two processing operations, the (a) processing andthe (b) processing, are exchangeably executed depending on whether ornot the interpolation W pixel is saturated.

By performing such processing, the compensation based on the saturatedinterpolation W pixel can be avoided and the high-precision compensation(NR) processing can be executed.

The noise reduction (NR) processing described above is processing thatuses a two-dimensional (2D) plane on which the reference area is set tobe the 7×7 pixels, but if the processing that uses the two-dimensionalplane is performed, it is necessary to retain at least a two-dimensional(2D) image for the reference area in the memory.

The image values from the imaging element are sequentially output lineby line in a horizontal direction (x-direction). That is, the imagevalues from the imaging element are sequentially input, asone-dimensional (1D) data, into the signal processing unit.

If the processing that uses the two-dimensional (2D) plane on which thereference area described above is set to be the 7×7 pixels is performed,there occurs a problem in that it is necessary to retain data for atleast 7 lines, a high-capacity memory is necessary, and for example,cost of a camera and the like are increased.

In order to avoid the use of the high-capacity memory, a configurationmay be possible in which using one-dimensional (1D) data line by line,the assigning of the interpolation W pixel described above is performedand the pixel value compensation as the noise reduction processing isperformed. Moreover, thereafter, for example, a configuration is alsopossible in which the compensation that refers to the pixel in thevertical direction is performed using an infinite impulse response (IIR)filter. Moreover, the IIR filter is for filter processing that isperformed on the signal that is input in a time-series, as processingthat refers to only a current signal and an earlier signal, and makes itpossible to omit a memory that retains all the reference pixel values.

FIG. 8 is a diagram for describing a processing example in which theassigning and the compensation of the interpolation W pixel describedabove are executed using the one-dimensional (1D) data line by line.

Data illustrated in FIG. 8(1) is equivalent to pixel data in thehorizontal line that is output from the imaging element.

At this point, the attention pixel that is set to be the noise reductionprocessing target is set to be the B pixel in the center portionillustrated in FIG. 8(1), which is an NR processing pixel. The pixelvalue of the NR processing pixel is I(p).

The interpolation W pixel is assigned to positions of the NR processingpixel and the B pixel that are included in the line.

The pixel value of the interpolation W pixel may be determined byapplying the pixel value of the adjacent W pixel, such as an averagevalue of the two W pixels that are adjacent to each B pixel.

After the interpolation W pixel assignment processing, the calculationof the compensation pixel value according to (Equation 1) describedabove is executed.

However, if the pixel value of the interpolation W pixel is saturated,the calculation of the compensation pixel value according to (Equation2) described above is executed without using the interpolation W pixel.

By performing such processing, the noise reduction processing with alarger number of TAPs is realized at a low cost.

A processing sequence of the noise reduction (NR) processing that isexecuted in the image processing apparatus according to the presentdisclosure is described referring to a flow chart that is illustrated inFIG. 9.

The flow chart that is illustrated in FIG. 9 illustrates processing thatis executed in a signal processing unit that executes signal processingby inputting a captured-image signal from the imaging element, forexample, in the image processing apparatus such as the digital camera.For example, the processing is executed in a data processing unit, suchas a CPU within the signal processing unit that has a function ofexecuting a program according to the program recorded in the signalprocessing sequence that is illustrated in FIG. 9.

The signal processing unit selects the pixel signals of RGBW that areinput from the imaging element, sequentially one by one as thecompensation target, and thus performs the processing.

First, in Step S101, it is determined whether or not the compensationtarget pixel (attention pixel) is a White (W) pixel.

If the compensation target pixel (attention pixel) is the White (W)pixel, proceeding to Step S107 takes place.

If the compensation target pixel (attention pixel) is not the White (W)pixel, that is, if the compensation target pixel (attention pixel) isany color pixel of the R, G, and B, proceeding to Step S102 takes place.

If the compensation target pixel (attention pixel) is the White (W)pixel, proceeding to Step S107 takes place, the smoothing processing inthe related art is executed, the interpolation pixel value of the Wpixel that is the interpolation target element (attention pixel) iscalculated, and thus the compensation target pixel is set to be thenoise reduction pixel.

In the RGBW arrangement, because the W pixel is high in pixel density,even though only the W pixel is selected as the reference pixel, thepixel value of the reference pixel that is made from only the W pixel isused, the smoothing weight is assigned, and thus the interpolation pixelvalue is calculated, there is a low likelihood that the compensationprecision will be decreased.

On the other hand, if the compensation target pixel (attention pixel) isnot the white (W), that is, if the compensation target pixel is anycolor pixel of the R, G, and B, proceeding to Step S102 takes place.

In Step S102, the W pixel interpolation processing is executed thatassigns the W pixel to a position of the reference pixel which has thesame color as the attention pixel that is in the reference pixel area ofwhich the center is the attention pixel.

Such processing is the W pixel interpolation processing that isdescribed above referring to FIGS. 3 and 5.

Next, in Step S103, it is determined whether or not even one saturationpixel is present in the interpolation W pixel.

If it is confirmed that even one saturation pixel is present in theinterpolation W pixel, proceeding to Step S105 takes place.

If it is confirmed that not even one saturation pixel is present in theinterpolation W pixel, proceeding to Step S104 takes place.

When it is confirmed that not even one saturation pixel is present inthe interpolation W pixel and proceeding to Step S104 takes place, theprocessing that calculates the interpolation pixel value according to(Equation 1) described above is performed in Step S104 and S106. Thatis, the pixel value of the attention pixel is calculated according to(Equation 1), using the smoothing weight to which the pixel value of theinterpolation W pixel is applied. The interpolation pixel value is setto be the pixel value of the attention pixel that results after thenoise reduction.

On the other hand, if it is confirmed that one or more saturation pixelsare present in the interpolation W pixel, in Steps S105 and S106, theprocessing that calculates the interpolation pixel value according to(Equation 2) described above is performed. That is, the pixel value ofthe attention pixel is calculated according to (Equation 2), bycalculating the smoothing weight to which the pixel values of theattention pixel and of the reference pixel which has the same color asthe attention pixel are applied. The interpolation pixel value is set tobe the pixel value of the attention pixel that results after the noisereduction.

In Step S106, when the calculation of the compensation pixel value towhich (Equation 1) or (Equation 2) is applied is ended, proceeding toStep S110 takes place.

In Step S110, it is determined whether or not the calculation of theinterpolation pixel value for all the pixels that make up the image isended. If a non-processed pixel is present, returning to Step S101 takesplace, processing in Step S101 and later is performed on thenon-processed pixel, and thus compensation pixel value calculationprocessing is performed on the non-processed pixel.

In Step S110, when it is determined that the processing on all thepixels is ended, the processing is ended.

[3. Example of Processing that Performs Defect Compensation According toa Result of Texture Detection to which the W Pixel is Applied inCompensation Processing of the Color Pixel that is a Defect CompensationTarget]

Next, as an example of processing that compensates a defective pixelthat is included in the image, which is executed in the image processingapparatus according to the present disclosure, an example of processingis described in which texture detection to which the W pixel in theneighborhood of the pixel position of the color pixel that is thecompensation target is applied is performed, it is determined whether ornot the compensation processing is executed according to the result ofthe texture detection, and thus the compensation is performed.

First, as described in FIGS. 1 and 2, since a distance between the colorpixels in the RGBW arrangement that includes the white (W) pixel isgreater than in the Bayer arrangement in the related art, there is aproblem in that the defect compensation is difficult to execute.

Particularly, if the defect compensation processing to which theone-dimensional (1D) pixel data is applied is performed as firstdescribed above in FIG. 8 in order to cut down on a hardware (HW) cost,because the distance between the pixels is greater, such a problem ispronounced.

Moreover, the defect compensation processing to which theone-dimensional (1D) pixel data is applied is, for example, processingthat selects as the reference pixel the neighborhood pixel that has thesame color as the pixel (attention pixel) that is the compensationtarget, applies the pixel value of the reference pixel, and thuscalculates the pixel value of the compensation pixel.

In this case, if the number of the pixels that have the same color andthat are present in the neighborhood of the compensation target pixel islarge, higher-precision compensation is possible. However, if the numberof the pixels that have the same color and that are available forreference in the neighborhood of the compensation target pixel is small,the precision of the defect compensation is decreased. Particularly, ifthe texture, such as a pattern of a photographic subject that isincluded in the captured image, is present, when damaged pixels arescattered, it is difficult to determine whether the defect or thetexture is present and the high-precision defect compensation isextremely difficult to execute.

For example, as illustrated in FIG. 10, in the RGBW arrangements thatare illustrated in FIGS. 10( a) to 10(d), when viewed from thehorizontal direction, whereas one W pixel out of every two pixels ispresent in all the arrangements, one R pixel and one B pixel out ofevery four pixels are present in all the arrangements and only one Gpixel out of every four pixels is present in the arrangements other thanthe arrangement in FIG. 10( c).

Because the greater the distance between the pixels that have the samecolor, the more difficult it is to separate the texture and the defect,in the related art, a technique of the one-dimensional defectcompensation that is performed many times cannot be applied as is.

An example of processing is described below that executes analysisprocessing that uses the white (W) pixel adjacent to the compensationtarget pixel (attention pixel), determines whether the defect or thetexture is present, and thus executes the defect compensation if it isdetermined that the defect is present.

First, an outline of the defect compensation processing according to thepresent disclosure is described referring to FIG. 11.

In a sensor that has been used in recent years, the pixel density ishigh, and it is observed that one bright dot on an actual scene in imagecapture processing extends over two or more pixels due tocharacteristics of an optical system. As illustrated in FIG. 11(1), itis observed that one bright dot, for example, extends not only over thecentral color pixel G, but also over the adjacent W pixel. On the otherhand, because optical characteristics do not have an influence on apixel defect, the signal level goes up and down greatly in one pixelalone.

Moreover, as illustrated in FIG. 11(2), because the white (W) pixel hascharacteristics that allow all-wavelength visible light to pass through,for example, if the signal level of the color pixel (G) that is thecompensation target pixel (attention pixel) in the center of thereference area goes up due to the influence of the bright dot, thesignal level of the adjacent white (W) pixel goes up.

However, if the pixel level of the color pixel (G) that is thecompensation target pixel (attention pixel) goes up due to the defect,the pixel level of the adjacent W pixel is comparatively lower comparedto the G pixel.

In other words, by performing the analysis processing that uses thewhite (W) pixel that is adjacent to the compensation target pixel(attention pixel), the determination of whether the defect is present orthe texture according to the captured image is present is effective.

Next, an example of detection processing of the general pixel defect isdescribed referring to FIGS. 12 and 13.

First, an example 1 of the detect detection processing is describedreferring to FIG. 12.

The example of the defect detection processing that is illustrated inFIG. 12 is an example of the defect detection processing that uses thesame one-dimensional (1D) pixel line data as described above referringto FIG. 8.

It is determined whether or not the compensation target pixel (attentionpixel) G(x) in the center of the pixel line in FIG. 12 is the defectivepixel.

In such processing, the pixel value of the neighborhood G pixel that hasthe same color as G(x) is obtained.

In the example on the drawings, the pixel value of each G pixel ofG(x+8), G(x+4), G(x), G(x−4), and G(x−8) is obtained.

Moreover, a maximum value (max) and a minimum value (min) among thepixel values of the five G pixels are determined.

If G(x) is the maximum (max), it is determined that there is alikelihood that the G(x) pixel will have a white dot defect, and if G(x)is the minimum (min), it is determined that the G(x) pixel will have ablack dot defect.

The white dot defect is a defect that results in outputting the greaterpixel value than the normal pixel value, and the black dot defect is adefect that results in outputting the smaller pixel value than thenormal pixel value.

Next, an example 2 of the detect detection processing is describedreferring to FIG. 13.

The example of the defect detection processing that is illustrated inFIG. 13 is processing that determines whether or not the compensationtarget pixel (attention pixel) G(x) is the defective pixel by comparing(a) the pixel value of the compensation target pixel (attention pixel)G(x) and (b) the estimation pixel value of the position of thecompensation target pixel (attention pixel) G(x) that is estimated fromthe pixel adjacent to the compensation target pixel (attention pixel)G(x).

Also in such processing, first, the pixel value of the neighborhood Gpixel that has the same color as G(x) is first obtained in the samemanner as in the example 1 of the processing described above.

In the example in the drawings, the pixel value of each G pixel ofG(x+8), G(x+4), G(x), G(x−4), and G(x−8) is obtained.

Moreover, an estimation pixel value GI(x) of the position of thecompensation target pixel (attention pixel) G(x) that is estimated fromthe two G pixels on the left side of G(x), that is, G(x−4) and G(x−8) iscalculated according to Equation described below.GI(x)=G(x−4)+{G(x−4)−G(x−8)}

In the same manner, an estimation pixel value Gr(x) of the position ofthe compensation target pixel (attention pixel) G(x) that is estimatedfrom the two G pixels on the right side of G(x), that is, G(x+4) andG(x+8) is calculated according to Equation described below.Gr(x)=G(x+4)+{G(x+4)−G(x+8)}

The two estimation pixel values, that is, GI(x) and Gr(x), and the pixelvalue G(x) of the actual compensation target pixel (attention pixel) arecompared.

If G(x) is greater than the maximum value (max) among the two estimationpixel values, that is, GI(x) and Gr(x), it is determined that there is alikelihood that a G(x) pixel will have the white dot defect, and if G(x)is smaller than the minimum value (min) among the two estimation pixelvalues, that is, GI(x) and Gr(x), it is determined that there is alikelihood that a G(x) pixel will have the black dot defect.

The white dot defect is a defect that results in outputting the greaterpixel value than the normal pixel value, and the black dot defect is adefect that results in outputting the smaller pixel value than thenormal pixel value.

The defect detection of the pixel, for example, is executed according tothe processing described referring to FIGS. 12 and 13. Moreover, inaddition, there is a variety of defect detection processing, and in theconfiguration according to the present disclosure, it is possible toapply the variety of defect detection processing.

Next, one example of processing that detects the texture from the image,which is performed in the image processing apparatus according to thepresent disclosure, is described referring to FIG. 14.

FIG. 14 illustrates (1) an example of assigning the pixel for thetexture detection and (2) a texture detection sequence.

(1) The example of assigning the pixel for the texture detection is anexample of using the one-dimensional (1D) pixel data, that is, the pixeldata in one line in the same manner as described above referring to FIG.8.

The compensation target pixel (attention pixel) that is the target fordetermining the presence and absence of the texture is set to be G(x)that is illustrated in FIG. 14(1).

The pixel value of the W pixel in the neighborhood of the attentionpixel G(x) is obtained.

In an example that is illustrated in FIG. 14(1), the pixel values of thefour neighborhood W pixels, that is, W(x−3), W(x−1), W(x+1), and W(x+3)are obtained.

For example, like in a graph that is illustrated in FIG. 14(1), thesignal level of each pixel that corresponds to each coordinate positionis obtained.

FIG. 14(2) illustrates processing that determines whether or not thereis a convex texture, that is, a texture in which the signal level of theattention pixel G(x) is higher than that of the adjacent pixels.

Because the pixel values of the multiple adjacent pixels, as describedabove referring to FIG. 11, are influenced according to one bright dot,in a case of the convex texture, it may be considered that the level ofthe adjacent W pixel also goes up.

Therefore, first, the maximum value of the W pixels adjacent to theattention pixel G(x) is calculated according to (Equation 3) describedbelow.Max (W(x−1), W(x+1))   (Equation 3)

Next, the minimum value of the two W pixels outside of the adjacent Wpixel described above is calculated according to (Equation 4) describedbelow.Min(W(x−3), W(x+3))  (Equation 4)

Next, a difference is calculated between the maximum value of the Wpixels adjacent to the attention pixel G(x), which is calculatedaccording to (Equation 3) described above, and the minimum value of theW pixels outside of the attention pixel G(x), which is calculatedaccording to (Equation 4), and thus is compared with a predeterminedthreshold (Th).

If the difference is greater than the threshold (Th), it is determinedthat the attention pixel G(x) is in the texture (convex texture).

If the difference is not greater than the threshold, it is determinedthat the attention pixel G(x) is not in the texture (convex texture).

That is, if(Max(W(x−1),W(x+1))−Min(W(x−3),W(x+3))>Th)  (Equation 5)

If (Equation 5) described above is valid, it is determined that theattention pixel G(x) is in the texture (convex texture).

If (Equation 5) described above is not valid, it is determined that theattention pixel G(x) is not in the texture (convex texture).

FIG. 14 (2) illustrates processing that determines whether or not theconvex texture, that is, the texture in which the signal level of theattention pixel G(x) is higher than that of the adjacent pixels, but inFIG. 14(2), processing is also executed that determines whether or notthere is a concave texture, that is, the texture in which the signallevel of the attention pixel G(x) is lower than that of the adjacentpixels.

In concave texture determination processing, first, the minimum value ofthe W pixels adjacent to the attention pixel G(x) is calculatedaccording to (Equation 3) described below.Min(W(x−1), W(x+1))  (Equation 6)

Next, the maximum value of the two W pixels outside of the adjacent Wpixels described above is calculated according to (Equation 7).Max(W(x−3), W(x+3))  (Equation 7)

Next, a difference is calculated between the minimum value of the Wpixels adjacent to the attention pixel G(x), which is calculatedaccording to (Equation 6) described above, and the maximum value of theW pixels outside of the attention pixel G(x), which is calculatedaccording to (Equation 7), and thus is compared with a predeterminedthreshold (Th).

If the difference is greater than the threshold (Th), it is determinedthat the attention pixel G(x) is in the texture (concave texture).

If the difference is not greater than the threshold, it is determinedthat the attention pixel G(x) is not in the texture (concave texture).

That is, if(Max(W(x−3),W(x+3))−Min(W(x−1),W(x+1))>Th)  (Equation 8)

If (Equation 8) described above is valid, it is determined that theattention pixel G(x) is in the texture (concave texture).

If (Equation 8) described above is not valid, it is determined that theattention pixel G(x) is not in the texture (concave texture).

In this manner, the image processing apparatus according to the presentdisclosure performs the texture determination processing to which the Wpixel is applied. If the processing that determines the likelihood ofthe defect, which is described referring to FIGS. 12 and 13, isexecuted, and furthermore it is determined that there is a likelihoodthat the defect will be present, the texture determination processingdescribed referring to FIG. 14 is performed. If it is determined thatthe pixel determined as having the likelihood of the defect is in thetexture, the pixel value, as is, is output as the effective pixel valuewithout determining that the defect is not present and thus executingthe pixel value compensation as the defect compensation.

A processing sequence for the defect compensation processing that isexecuted in the image processing apparatus according to the presentdisclosure is described referring to FIG. 15.

The flow chart that is illustrated in FIG. 15 illustrates processingthat is executed in a signal processing unit that executes signalprocessing by inputting a captured-image signal from the imagingelement, for example, in the image processing apparatus such as thedigital camera. For example, the processing is executed in a dataprocessing unit, such as a CPU within the signal processing unit thathas a function of executing a program according to the program recordedin the signal processing sequence that is illustrated in FIG. 15.

The signal processing unit selects the pixel signals of RGBW that areinput from the imaging element, sequentially one by one as thecompensation target, and thus performs the processing.

First, in Step S201, it is determined whether or not the compensationtarget pixel (attention pixel) is the white (W) pixel.

If the compensation target pixel (attention pixel) is the white (W)pixel, proceeding to Step S207 takes place.

If the compensation target pixel (attention pixel) is not the White (W)pixel, that is, if the compensation target pixel (attention pixel) isany color pixel of the R, G, and B, proceeding to Step 202 takes place.

If the compensation target pixel (attention pixel) is the white (W)pixel, proceeding to Step S207 takes place, and the defect compensationprocessing in the related art is executed. In Step S207, it isdetermined whether there is a likelihood that the pixel will bedefective, and if it is determined that the defective pixel is present,the compensation pixel value of the W pixel that is the compensationtarget pixel (attention pixel) is calculated. Then, the compensationpixel value is set to be an output pixel. Moreover, because the pixeldensity of the W pixels is high in the RGBW arrangement, even though thecompensation value is calculated by selecting only the W pixel as thereference pixel and thus using the pixel value of the reference pixelthat is made from only the W pixel, there is a low likelihood that thecompensation precision will be decreased.

On the other hand, if the compensation target pixel (attention pixel) isnot the white (W) pixel, that is, if the compensation target pixel(attention pixel) is any color pixel of the R, G, and B, proceeding toStep S202 takes place.

In Step S202, it is determined whether there is a likelihood that theattention pixel that is any color pixel of the R, G, and B will be thedefective pixel.

Such processing is executed by applying, for example, the defectdetection processing that is described above referring to FIGS. 12 and13.

In Step S203, if it is determined that there is a likelihood that theattention pixel which is any color pixel for the R, G, and B will be thedefective pixel, proceeding to Step S204 takes place.

On the other hand, in Step S203, if it is determined that there is nolikelihood that the attention pixel which is any color pixel for the R,G, and B will be the defective pixel, proceeding to Step S210 takesplace without performing the defect compensation on the attention pixel.

In Step S203, if it is determined that there is a likelihood that theattention pixel which is any color pixel for the R, G, and B will be thedefective pixel, in Step S204, furthermore, the texture determinationprocessing is performed that determines whether or not the attentionpixel is in a texture area.

The texture determination processing is executed by applying the W pixelin the neighborhood of the attention pixel. That is, the texturedetermination processing described above referring to FIG. 14 isexecuted.

In Step S205, if it is determined that the attention pixel is in thetexture area, proceeding to Step S210 takes place without performing thedefect compensation processing on the attention pixel.

On the other hand, in Step S205, if it is determined that the attentionpixel is not in the texture area, it is determined that the attentionpixel is the defective pixel, and in Step S206, the defect compensationprocessing is executed.

Moreover, the defect compensation processing, for example, performs thefollowing processing.

If there is a likelihood that the attention pixel will be the white dot,and the attention pixel is not the convex texture, for example, anyprocessing of the following defect compensation (a) and (b) isperformed.

(a) Among the pixel values of the four pixels in the neighborhood of theattention pixel, which have the same color, the second pixel value(2nd_max) in the increasing order of the pixel value is set to be thecompensation pixel value of the attention pixel.

(b) The maximum value, among a first estimation pixel value from thepixel value of the left-side pixel that has the same color as theattention pixel, which is described above referring to FIG. 13, and asecond estimation pixel value from the pixel value of the right-sidepixel that has the same color, is set to be the compensation pixel valueof the attention pixel.

For example, if the attention pixel is G(x) that is illustrated in FIG.13, the processing in (b) described above assigns the compensation pixelvalue as follows.

According to the following Equation, the first estimation pixel valueGI(x) is calculated from the two G pixels, that is, G(x−4) and G(x−8),on the left side of G(x).GI(x)=G(x−4)+{G(x−4)−G(x−8)}

Moreover, according to Equation described below, the second estimationpixel value Gr(x) is calculated from the two G pixels, that is, G(x+4)and G(x+8) on the right side of G(x).Gr(x)=G(x+4)+{G(x+4)−G(x+8)}

The maximum value, among two estimation pixel values of these: GI(x) andGr(x), that is, max (GI(x), Gr(x)) that is the maximum pixel value thatis any one of the GI(x) and Gr(x) that are selected according toEquations described above is set to be the compensation pixel value ofthe attention pixel G(x) that is the compensation target.

The compensation pixel value is assigned by such processing.

If there is a likelihood that the attention pixel will be the black dot,and the attention pixel is not in the concave texture, for example, anyprocessing of the following defect compensation (c) and (d) isperformed.

(c) Among the pixel values of the four pixels in the neighborhood of theattention pixel, which have the same color, the second pixel value(2nd_min) in the decreasing order of the pixel value is set to be thecompensation pixel value of the attention pixel.

(d) The minimum value, among a first estimation pixel value from thepixel value of the left-side pixel that has the same color as theattention pixel, which is described above referring to FIG. 13, and asecond estimation pixel value from the pixel value of the right-sidepixel that has the same color, is set to be the compensation pixel valueof the attention pixel.

For example, if the attention pixel is G(x) that is illustrated in FIG.13, the processing in (d) described above assigns the compensation pixelvalue as follows.

According to the following Equation, the first estimation pixel valueGI(x) is calculated from the two G pixels, that is, G(x−4) and G(x−8),on the left side of the G(x).GI(x)=G(x−4)+{G(x−4)−G(x−8)}

Moreover, according to Equation described below, the second estimationpixel value Gr(x) is calculated from the two G pixels, that is, G(x+4)and G(x+8) on the right side of the G(x).Gr(x)=G(x+4)+{G(x+4)−G(x+8)}

The minimum value, among two estimation pixel values of these: GI(x) andGr(x), that is, min (GI(x), Gr(x)) that is the minimum pixel value thatis any one of the GI(x) and Gr(x) that are selected according toEquations described above is set to be the compensation pixel value ofthe attention pixel G(x) that is the compensation target.

The compensation pixel value is assigned by such processing.

The defect compensation processing in Step S206 is executed as suchcompensation pixel value assignment processing of the attention pixel.

Next, in Step S210, it is determined whether or not the defectcompensation processing in Steps 201 to 207 is performed on all thepixels that make up the image. If a non-processed pixel is present,returning to Step S201 takes place, and the processing is performed onthe non-processed pixel by executing the processing in Step S201 andlater.

In Step S210, when it is determined that the processing on all thepixels is ended, the processing is ended.

[4. Configuration Example of an Imaging Processing Apparatus]

Next, a configuration example of the image processing apparatus thatexecutes the noise reduction processing and the defect compensationprocessing that are described above is described.

Multiple configuration examples of the image processing apparatusaccording to the present disclosure are described referring FIGS. 16 to18. The signal processing units that are described in FIGS. 16 to 18 areconfigured, for example, as signal processing units in the digitalcamera. FIGS. 16 to 18 are equivalent to signal processing units thatare set as follows.

(a) A signal processing unit 200 that is illustrated in FIG. 16: asignal processing unit that executes the defect compensation processingthat uses the one-dimensional (1D) pixel data and the noise reduction(NR) processing that uses the reference area that is a two-dimensional(2D) area.

(b) A signal processing unit 300 that is illustrated in FIG. 17: asignal processing unit that executes the defect compensation processingthat uses the one-dimensional (1D) pixel data and the noise reduction(NR) processing.

(c) A signal processing unit 400 in FIG. 18: a signal processing unitthat executes the defect compensation processing that uses the referencearea that is the two-dimensional (2D) area and the noise reduction (NR)processing.

Configurations of and processing by such signal processing units aresequentially described below.

Moreover, any one of the signal processing units is configured to be,for example, within the digital camera, and for example, according to aprogram that is stored in a memory in the digital camera, inputs acontrol signal from a control unit that is configured from a CPU and thelike, and according to timing or a sequence that is stipulated by thecontrol signal, executes stipulated processing sequentially.

[4-1. Configuration Example 1 of the Image Processing Apparatus]

First, as a first configuration example of the image processingapparatus, an example of an image processing apparatus, which has thesignal processing unit 200 that executes the defect compensationprocessing that uses the one-dimensional (1D) pixel data and the noisereduction (NR) processing that uses the reference area that is thetwo-dimensional (2D) area, is described referring to FIG. 16.

The signal processing unit 200 that is illustrated in FIG. 16 has a dataconversion processing unit 210 and an RGB signal processing unit 230.

The data conversion processing unit 210, as illustrated in FIG. 16,sequentially inputs the pixel signals from an imaging element (imagesensor) 150 that has the RGBW pixel arrangement, in the sequence of thepixels in the line in the horizontal direction.

The data conversion processing unit 210 selects the compensation targetpixel in the sequence of the pixels in the line in the horizontaldirection that are input from the imaging element (image sensor) 150 andthus executes the defect compensation processing that uses theone-dimensional (1D) pixel data and the noise reduction (NR) processingthat uses the reference area that is the two-dimensional (2D) area.

First, in the W pixel defect compensation unit 211 and the color pixeldefect compensation unit 212, the defect compensation processing isexecuted that is described referring to FIGS. 10 to 15, with regard tothe item described above, that is, [3. Example of Processing thatPerforms Defect Compensation according to a Result of Texture Detectionto which the W pixel is applied in Compensation Processing of the ColorPixel that is a Defect Compensation Target].

The W pixel defect compensation unit 211 executes the processing if thedefect compensation target pixel (attention pixel) is the W pixel. Thecolor pixel defect compensation unit 212 executes the processing if thedefect compensation target pixel (attention pixel) is any color pixel ofthe R, G, and B other than the W pixel.

Moreover, in such processing, the color pixel defect compensation unit212 performs the texture detection based on the W pixel as describedabove, and if it is determined that the attention pixel is in thetexture area, outputs the original pixel value, as is, as the effectivepixel value without performing the pixel value compensation of theattention pixel. If it is determined that the attention pixel is thedefective pixel, but is not in the texture, the compensation isexecuted, and thus the compensation pixel value is assigned.

In the W pixel defect compensation unit 211 and the color pixel defectcompensation unit 212, the pixel data, the defect compensationprocessing on which is ended, is stored in a line memory 213.

A next W pixel noise reduction (NR) processing unit 214, and a colorpixel noise reduction (NR) processing unit 215 execute the noisereduction processing that sets the reference area that is thetwo-dimensional area, using the image data that is stored in the linememory 213. Such processing is the processing that is describedreferring to FIGS. 2 to 9 with regard to the item described above, thatis, [2. Example of Processing that Calculates a Compensation Pixel Valueof a Color Pixel by Interpolating a W Pixel at a Pixel Position of theColor Pixel that is a Compensation Target and Applying the InterpolationW Pixel].

For example, the noise reduction processing is performed that isdescribed with regard to the item [2] described above, by setting thereference area, such as the 7×7 pixels of which the center is set to bethe attention pixel that is set to be the processing target.

The W pixel noise reduction (NR) processing unit 214 executes theprocessing if a noise reduction processing target pixel (attentionpixel) is the W pixel. The color pixel noise reduction (NR) processingunit 215 executes the processing if the noise reduction processingtarget pixel (attention pixel) is any color pixel of the R, G, and Bother than the W pixel.

Moreover, in such processing, the color pixel noise reduction (NR)processing unit 215, as described above referring to FIGS. 3, 5, 7 andother figures, performs assigning of the interpolation W pixel to thepositions of the attention pixel and the reference pixel in thereference area, and calculates the smoothing weight to which theinterpolation W pixel is applied, and thus calculates the interpolationpixel value according to (Equation 1) described above.

However, if the interpolation W pixel is saturated, the compensationpixel value calculation processing to which (Equation 2) described aboveis applied is executed.

Moreover, the compensation pixel data on which each of the defectcompensation processing and the noise reduction processing is performedis input into a color correlation re-mosaic processing unit 220.

The color correlation re-mosaic processing unit 220 inputs an RGBWsignal that is an output signal from the W pixel noise reduction (NR)processing unit 214 and the color pixel noise reduction (NR) processingunit 215, and executes processing for conversion from an RGBW colorarrangement to an RGB arrangement 231.

Specifically, for example, five types of conversion or compensationprocessing as are described below are executed.

Convert a position of the W pixel into the G pixel (Estimate the G pixelvalue)=(GonW)

Convert a position of the G pixel into the R pixel (Estimate the R pixelvalue)=(RonG)

Convert the position of the G pixel into the B pixel (Estimate the Bpixel value)=(BonG)

Convert a position of the R pixel into the R pixel (Estimate the R pixelvalue)=(RonR)

Convert a position of the B pixel into the B pixel (Estimate the B pixelvalue)=(BonB)

Moreover, an aspect of such re-mosaic processing is one example, and theaspect of the re-mosaic processing is determined according to acorrespondence relationship between an input image signal that isdetermined by a configuration of the color filter that is set in theimaging element, and an output image signal that is output to the RGBsignal processing unit 230.

According to the present example, each constituent element of the colorcorrelation re-mosaic processing unit 220 executes the followingprocessing.

A W position G interpolation parameter calculation unit 221 calculatesthe interpolation parameter that is applied to calculation of the Gpixel value that is assigned to the position of the W pixel in the RGBWarrangement.

A G position RB interpolation parameter calculation unit 222 calculatesthe interpolation parameter that is applied to calculation of the Rpixel or the B pixel that is assigned to the position of the G pixel inthe RGBW arrangement.

A R position R interpolation parameter calculation unit 223 calculatesthe interpolation parameter that is applied to calculation of thecompensation R pixel value that is assigned to the position of the Rpixel in the RGBW arrangement.

A B position B interpolation parameter calculation unit 224 calculatesthe interpolation parameter that is applied to calculation of thecompensation B pixel value that is assigned to the position of the Bpixel in the RGBW arrangement.

A weight addition processing unit 225 inputs the interpolation parameterthat is calculated by each of the interpolation parameter calculationunits 221 to 224, and calculates the RGB signal value of each pixel thatmakes up the RGB arrangement (Bayer arrangement) 231.

Moreover, for the processing for data conversion from the RGBWarrangement to the RGB arrangement, which is executed by the colorcorrelation re-mosaic processing unit (data conversion unit) 220,basically, it is possible to use the processing that is disclosed inJapanese Unexampled Patent Application Publication No. 2011-55038 thatwas filed earlier by the applicant. Refer to Japanese Unexamined PatentApplication Publication No. 2011-55038 for details of the processing forthe data conversion processing.

In this manner, the RGB arrangement (Bayer arrangement) 231 that isgenerated by the weight addition processing unit 225 is output to theRGB signal processing unit 230.

The RGB signal processing unit 230 is the same as a signal processingunit to an RGB arrangement (Bayer arrangement) signal, which a generalcamera or image processing apparatus has. The RGB signal processing unit230 generates the color image by executing the signal processing on theRGB arrangement (Bayer arrangement) 231 that is output from the weightaddition processing unit 225. Specifically, the RGB signal processingunit 230 generates the color image by executing, for example, whitebalance adjustment processing, de-mosaic processing, shading processing,RGB color matrix processing, γ correction processing, and the like.

[4-2. Configuration Example 2 of the Image Processing Apparatus]

Next, as a configuration example of a second image processing apparatus,an example of an image processing apparatus that has the signalprocessing unit 300 that executes the defect compensation processingthat uses the one-dimensional (1D) pixel data and the noise reduction(NR) processing is described referring to FIG. 17.

The signal processing unit 300 that is illustrated in FIG. 17 has a dataconversion processing unit 310 and an RGB signal processing unit 330.

The data conversion processing unit 310, as illustrated in FIG. 17,sequentially inputs the pixel signals from the imaging element (imagesensor) 150 that has the RGBW pixel arrangement, in the sequence of thepixels in the line in the horizontal direction.

The data conversion processing unit 310 selects the compensation targetpixel in the sequence of the pixels in the line in the horizontaldirection that are input from the imaging element (image sensor) 150 andthus executes the defect compensation processing that uses theone-dimensional (1D) pixel data and the noise reduction (NR) processing.

First, in the W pixel defect compensation unit 311 and the color pixeldefect compensation unit 312, the defect compensation processing isexecuted that is described referring to FIGS. 10 to 15, with regard tothe item described above, that is, [3. Example of Processing thatPerforms Defect Compensation according to a Result of Texture Detectionto which the W pixel is applied in Compensation Processing of the ColorPixel that is a Defect Compensation Target].

The W pixel defect compensation unit 311 executes the processing if thedefect compensation target pixel (attention pixel) is the W pixel. Thecolor pixel defect compensation unit 312 executes the processing if thedefect compensation target pixel (attention pixel) is any color pixel ofthe R, G, and B other than the W pixel.

Moreover, in such processing, the color pixel defect compensation unit312 performs the texture detection based on the W pixel as describedabove, and if it is determined that the attention pixel is in thetexture area, outputs the original pixel value, as is, as the effectivepixel value without performing the pixel value compensation of theattention pixel. If it is determined that the attention pixel is thedefective pixel, but is not in the texture, the compensation isexecuted, and thus the compensation pixel value is assigned.

In the W pixel defect compensation unit 311, the pixel data, the defectcompensation processing on which is ended, is stored in the line memory213.

In the color pixel defect compensation unit 312, the pixel data, thedefect compensation processing on which is ended, is input into a colorpixel horizontal noise reduction (NR) processing unit 313.

The color pixel horizontal noise reduction (NR) processing unit 313sequentially inputs the pixel data, the defect compensation processingon which is ended in the color pixel defect compensation unit 312, asone-dimensional (1D) pixel line data, and executes the noise reductionprocessing that uses the one-dimensional (1D) pixel line data. Suchprocessing is the processing that is described referring to FIG. 8 withregard to the item described above, that is, [2. Example of Processingthat Calculates a Compensation Pixel Value of a Color Pixel byInterpolating a W Pixel at a Pixel Position of the Color Pixel that is aCompensation Target and Applying the Interpolation W Pixel].

As described referring to FIG. 8, the W pixel is interpolated at theposition of the color pixel that is the noise reduction processingtarget and the position of the neighborhood pixel of the same color inthe pixel line, the assigning of the interpolation W pixel is performed,the smoothing weight to which the interpolation W pixel is applied iscalculated, and thus the compensation pixel value is calculatedaccording to (Equation 1) described above.

However, if the interpolation W pixel is saturated, the compensationpixel value calculation processing to which (Equation 2) described aboveis applied is executed.

The pixel data in which the noise is reduced by applying a horizontalline is input into a color pixel vertical noise reduction (NR) unit 314.

The color pixel vertical noise reduction (NR) unit 314, for example,performs the compensation that refers to the pixel in the verticaldirection, using the infinite impulse response (IIR) filter.

The output of the color pixel vertical noise reduction (NR) unit 314 isinput into a line memory 315.

Moreover, the compensation pixel data on which each of the defectcompensation processing and the noise reduction processing is performed,which is stored in the line memory 315, is input into a colorcorrelation re-mosaic processing unit 320.

The color correlation re-mosaic processing unit 320 has a W position Ginterpolation parameter calculation unit 321, a G position RBinterpolation parameter calculation unit 322, an R position Rinterpolation parameter calculation unit 323, a B position Binterpolation parameter calculation unit 324, and a weight additionprocessing unit 325.

The color correlation re-mosaic processing unit 320, like the colorcorrelation re-mosaic processing unit 220 described above referring toFIG. 16, executes the processing for the conversion from the RGBW colorarrangement to an RGB arrangement 331, and thus outputs the generatedRGB arrangement 331 to the RGB signal processing unit 330.

The RGB signal processing unit 330 is the same as a signal processingunit that performs the processing on the RGB arrangement (Bayerarrangement) signal, which is installed within a general camera or imageprocessing apparatus. The RGB signal processing unit 330 generates thecolor image by executing the signal processing on the RGB arrangement(Bayer arrangement) 331 that is output from the weight additionprocessing unit 325. Specifically, the RGB signal processing unit 330generates the color image by executing, for example, the white balanceadjustment processing, the de-mosaic processing, the shading processing,the RGB color matrix processing, the γ correction processing, and thelike.

[4-3. Configuration Example 3 of the Image Processing Apparatus]

Next, referring to FIG. 18, as a configuration example of a third imageprocessing apparatus, an example of an image processing apparatus isdescribed that has a signal processing unit 400 which executes thedefect compensation processing that uses the reference area that is thetwo-dimensional (2D) area and the noise reduction (NR) processing.

The signal processing unit 400 that is illustrated in FIG. 18 has a dataconversion processing unit 410 and an RGB signal processing unit 430.

The data conversion processing unit 410, as illustrated in FIG. 18,sequentially inputs the pixel signals from the imaging element (imagesensor) 150 that has the RGBW pixel arrangement, in the sequence of thepixels in the line in the horizontal direction.

The data conversion processing unit 410 sequentially stores the pixelsin the line in the horizontal direction, which are input from theimaging element (image sensor) 150, in a line memory 411, andthereafter, executes the defect compensation processing and the noisereduction (NR) processing using the two-dimensional (2D) image data thatis stored in the line memory 411.

First, in a W pixel defect compensation unit 412 and a color pixeldefect compensation unit 413, the defect compensation processing, whichis described referring to FIGS. 10 to 15, with regard to the itemdescribed above, that is, [3. Example of Processing that Performs DefectCompensation according to a Result of Texture Detection to which the Wpixel is applied in Compensation Processing of the Color Pixel that is aDefect Compensation Target], is executed on the image data stored in theline memory 411.

The W pixel defect compensation unit 412 executes the processing if thedefect compensation target pixel (attention pixel) is the W pixel. Thecolor pixel defect compensation unit 413 executes the processing if thedefect compensation target pixel attention pixel) is any color pixel ofthe R, G, and B other than the W pixel.

Moreover, in such processing, the color pixel defect compensation unit412 performs the texture detection based on the W pixel as describedabove, and if it is determined that the attention pixel is in thetexture area, outputs the original pixel value, as is, as the effectivepixel value without performing the pixel value compensation of theattention pixel. If it is determined that the attention pixel is thedefective pixel, but is not in the texture, the compensation isexecuted, and thus the compensation pixel value is assigned.

In the W pixel defect compensation unit 411 and the color pixel defectcompensation unit 412, the items of image data, the defect compensationprocessing on which are ended, are a next W pixel noise reduction (NR)processing unit 414 and a color pixel noise reduction (NR) processingunit 415, respectively.

A W pixel noise reduction (NR) processing unit 414 and a color pixelnoise reduction (NR) processing unit 415 execute the noise reductionprocessing that sets the reference area that is the two-dimensionalarea. Such processing is the processing that is described referring toFIGS. 2 to 9 with regard to the item described above, that is, [2.Example of Processing that Calculates a Compensation Pixel Value of aColor Pixel by Interpolating a W Pixel at a Pixel Position of the ColorPixel that is a Compensation Target and Applying the Interpolation WPixel].

For example, the noise reduction processing is performed that isdescribed with regard to the item [2] described above, by setting thereference area, such as the 7×7 pixels of which the center is set to bethe attention pixel that is set to be the processing target.

The W pixel noise reduction (NR) processing unit 414 executes theprocessing if the noise reduction processing target pixel (attentionpixel) is the W pixel. The color pixel noise reduction (NR) processingunit 415 executes the processing if the noise reduction processingtarget pixel (attention pixel) is any color pixel of the R, G, and Bother than the W pixel.

Moreover, in such processing, the color pixel noise reduction (NR)processing unit 415, as described above referring to FIGS. 3, 5, 7 andother figures, performs the assigning of the interpolation W pixel tothe positions of the attention pixel and the reference pixel in thereference area, and calculates the compensation pixel value according to(Equation 1) described above by calculating the smoothing weight towhich the interpolation W pixel is applied.

However, if the interpolation W pixel is saturated, the compensationpixel value calculation processing to which (Equation 2) described aboveis applied is executed.

Moreover, the compensation pixel data on which each of the defectcompensation processing and the noise reduction processing is performedis input into a color correlation re-mosaic processing unit 420.

The color correlation re-mosaic processing unit 420 has a W position Ginterpolation parameter calculation unit 421, a G position RBinterpolation parameter calculation unit 422, an R position Rinterpolation parameter calculation unit 423, a B position Binterpolation parameter calculation unit 424, and a weight additionprocessing unit 425.

The color correlation re-mosaic processing unit 420, like the colorcorrelation re-mosaic processing unit 220 described above referring toFIG. 16, executes the processing for the conversion from the RGBW colorarrangement to an RGB arrangement 431, and thus outputs the generatedRGB arrangement 431 to the RGB signal processing unit 430.

The RGB signal processing unit 430 is the same as a signal processingunit to the RGB arrangement (Bayer arrangement) signal, which isinstalled within a general camera or image processing apparatus. The RGBsignal processing unit 430 generates the color image by executing thesignal processing on the RGB arrangement (Bayer arrangement) 431 that isoutput from the weight addition processing unit 425. Specifically, theRGB signal processing unit 430 generates the color image by executing,for example, the white balance adjustment processing, the de-mosaicprocessing, the shading processing, RGB color matrix processing, the γcorrection processing, and the like.

[5. Conclusions of Configurations of the Present Disclosure]

The examples according to the present disclosure are described in detailabove referring to the specific examples. However, it is apparent that aperson of ordinary skill in the art can accomplish modifications to orsubstitutes for the examples in a range that does not deviate from thegist of the present disclosure. That is, because the present inventionis disclosed in the form of embodiments, the present invention shouldnot be interpreted in a limited manner. In order to determine the gistof the present disclosure, the scope of the claims should be considered.

Moreover, the technology disclosed in the present specification can beconfigured as follows.

(1) An image processing apparatus including: a signal processing unitthat executes pixel value compensation, in which the signal processingunit inputs image data in an RGBW arrangement that has each color pixelof R, G, and B and a white (W) pixel that passes through almost allwavelength light of each wavelength of the R, G, and B, and in which thesignal processing unit interpolates the W pixel at a position of anattention pixel that is a compensation target, and at a position of areference pixel which has the same color as the attention pixel within areference area, in a pixel value compensation processing of a colorpixel, calculates smoothing weight based on each pixel value of theinterpolation W pixel, and thus calculates a compensation pixel value ofthe attention pixel by executing smoothing processing to which thecalculated smoothing weight is applied.

(2) The image processing apparatus according to (1), in which the signalprocessing unit determines whether or not one or more saturation pixelvalues are present in the pixel values of the interpolation W pixel, inwhich if the saturation pixel value is not present in the pixel valuesof the interpolation W pixel, the signal processing unit calculates acompensation pixel value of the attention pixel by executing thesmoothing processing to which the smoothing weight, calculated based onthe each pixel value of the interpolation W pixel, is applied, and inwhich if the saturation pixel value is present in the pixel values ofthe interpolation W pixel, the signal processing unit calculates thecompensation pixel value of the attention pixel by executing thesmoothing processing to which the smoothing weight, calculated based oneach pixel value of the attention pixel that is the compensation target,and of the reference pixel which has the same color as the attentionpixel within the reference area, is applied without applying theinterpolation W pixel.

(3) The image processing apparatus according to (1) or (2), in which thesignal processing unit executes processing that interpolates the W pixelat the position of the reference pixel which has the same color as theattention pixel present in the reference area that is a two-dimensionalarea with the reference area as the two-dimensional area.

(4) The image processing apparatus according to any one of (1) to (3),in which the signal processing unit executes processing thatinterpolates the W pixel at the position of the reference pixel whichhas the same color as the attention pixel present in the reference areathat is a one-dimensional area with the reference area as theone-dimensional area.

(5) The image processing apparatus according to any one of (1) to (4),in which the signal processing unit executes the pixel valuecompensation as noise reduction (NR) processing that reduces noise thatis included in the attention pixel.

(6) The image processing apparatus according to any one of (1) to (5),in which the signal processing unit determines a likelihood of a defect,that is, determines whether or not there is a likelihood that the colorpixel will be a defective pixel, in which the signal processing unitexecutes texture detection processing that determines whether or not thecolor pixel is in a texture area, by applying the W pixel in theneighborhood of the color pixel that is determined as having thelikelihood of the defect, in which in the texture detection processing,if it is determined that the color pixel is in the texture area, thesignal processing unit does not execute defect compensation processing,and in which in the texture detection processing, if it is determinedthat the color pixel is not in the texture area, the signal processingunit executes the defect compensation processing.

(7) The image processing apparatus according to (6), in which in thetexture detection processing, the signal processing unit determineswhether or not the color pixel is in the texture area, by applying adifference in the pixel value between the W pixel that is closest to thecolor pixel that is determined as having the likelihood of the defect,and the W pixel outside of the closest W pixel.

Moreover, a method of implementing the processing that is executed inthe apparatus described above and a system, or a program that executesthe processing, and a recording medium on which the program is storedare included in the configuration according to the present disclosure.

Furthermore, it is possible to execute a sequence of the processingdescribed throughout the specification in hardware, software, or acombination configuration of both. If the processing is executed insoftware, a program in which the processing sequence is recorded isinstalled on a memory within a computer that is integrated intodedicated hardware and thus is executed, but alternatively it ispossible to install the program on an all-purpose computer that iscapable of executing a variety of processing and thus the program. Forexample, the program can be recorded in advance on a recording medium.In addition to installing the program on the computer from the recordingmedium, the program can be received over a network, such as a local areanetwork (LAN) or the Internet, and can be installed on the recordingmedium, such as built-in hardware.

Moreover, the variety of processing described in the specification isexecuted not only in a time series according to the description, but mayalso be executed in parallel or individually according to the processingcapability of an apparatus that executes the processing or whenevernecessary. Furthermore, the system in the present specification isconfigured to be a logical combination of multiple apparatuses, and theapparatuses in each configuration are not limited to being within thesame housing.

INDUSTRIAL APPLICABILITY

With the configuration of an example according to the presentdisclosure, as described above, the apparatus for and a method ofexecuting the noise reduction processing and the defect compensationprocessing on the image in the RGBW arrangement are realized.

Specifically, in the pixel value compensation processing of the colorpixel that makes up the image data in the RGBW arrangement that has eachcolor pixel of the R, G, and B and the white (W) pixel, the W pixel isinterpolated at the position of the attention pixel that is thecompensation target, and at the position of the reference pixel that isthe pixel that has the same color as the attention pixel within thereference area, the smoothing weight is calculated based on each pixelvalue of the interpolation W pixel, and thus the compensation pixelvalue of the attention pixel is calculated by executing the smoothingprocessing to which the calculated smoothing weight is applied.Moreover, by applying the W pixel in the neighborhood of the colorpixel, it is determined whether or not the color pixel is in the texturearea, and only if the color pixel is in the texture, the defectcompensation processing is executed.

With such processing, the apparatus for and the method of executing thenoise reduction processing and the defect compensation processing on theimage in the RGBW arrangement are realized.

REFERENCE SIGNS LIST

-   -   150 IMAGING ELEMENT, 200 SIGNAL PROCESSING UNIT, 210 DATA        CONVERSION PROCESSING UNIT, 211 W PIXEL DEFECT COMPENSATION        UNIT, 212 COLOR PIXEL DEFECT COMPENSASTION UNIT, 213 LINE        MEMORY, 214 W PIXEL NOISE REDUCTION (NR) UNIT, 215 COLOR PIXEL        NOISE REDUCTION (NR) UNIT, 220 COLOR CORRELATION RE-MOSAIC        PROCESSING UNIT, 221 W POSITION G INTERPOLATION PARAMETER        CALCULATION UNIT, 222 G POSITION RB INTERPOLATION PARAMETER        CALCULATION UNIT, 223 R POSITION R INTERPOLATION PARAMETER        CALCULATION UNIT, 224 B POSITION B INTERPOLATION PARAMETER        CALCULATION UNIT, 225 WEIGHT ADDITION UNIT, 230 RGB SIGNAL        PROCESSING UNIT, 231 RGB ARRANGEMENT, 300 SIGNAL PROCESSING        UNIT, 310 DATA CONVERSION PROCESSING UNIT, 311 W PIXEL DEFECT        COMPENSATION UNIT, 312 COLOR PIXEL DEFECT COMPENSATION UNIT, 313        COLOR PIXEL HORIZONTAL NOISE REDUCTION (NR) UNIT, 314 COLOR        PIXEL VERTICAL NOISE REDUCTION (NR) UNIT, 315 LINE MEMORY, 320        COLOR CORRELATION RE-MOSAIC PROCESSING UNIT, 321 W POSITION G        INTERPOLATION PARAMETER CALCULATION UNIT, 322 G POSITION RB        INTERPOLATION PARAMETER CALCULATION UNIT, 323 R POSITION R        INTERPOLATION PARAMETER CALCULATION UNIT, 324 B POSITION B        INTERPOLATION PARAMETER CALCULATION UNIT, 325 WEIGHT ADDITION        UNIT, 330 RGB SIGNAL PROCESSING UNIT, 331 RGB ARRANGEMENT, 400        SIGNAL PROCESSING UNIT, 410 DATA CONVERSION PROCESSING UNIT, 411        LINE MEMORY, 412 W PIXEL DEFECT COMPENSATION UNIT, 413 COLOR        PIXEL DEFECT COMPENSATION UNIT, 414 W PIXEL NOISE REDUCTION (NR)        UNIT, 415 COLOR PIXEL NOISE REDUCTION (NR) UNIT, 420 COLOR        CORRELATION RE-MOSAIC PROCESSING UNIT, 421 W POSITION G        INTERPOLATION PARAMETER CALCULATION UNIT, 422 G POSITION RB        INTERPOLATION PARAMETER CALCULATION UNIT, 423 R POSITION R        INTERPOLATION PARAMETER CALCULATION UNIT, 424 B POSITION B        INTERPOLATION PARAMETER CALCULATION UNIT, 425 WEIGHT ADDITION        UNIT, 430 RGB SIGNAL PROCESSING UNIT, 431 RGB ARRANGEMENT

The invention claimed is:
 1. An image processing apparatus comprisingcircuitry configured to: receive image data acquired by an image sensorin an RGBW arrangement that has each color pixel of R, G, and B and awhite (W) pixel that passes through almost all wavelength light of eachwavelength of the R, G, and B, interpolate W pixels at a position of anattention pixel that is a compensation target, and at a position of areference pixel which has the same color as the attention pixel within areference area, in a pixel value compensation processing of a colorpixel, calculate smoothing weight based on pixel values of theinterpolation W pixels, and thus calculate a compensation pixel value ofthe attention pixel by executing smoothing processing to which thecalculated smoothing weight is applied, wherein the smoothing processingcomprises computing a product of (a) a pixel value of the referencepixel which has the same color as the attention pixel and (b) adifference between the pixel value of the interpolation W pixel at theposition of the attention pixel and the pixel value of the interpolationW pixel at the position of the reference pixel, and provide thecompensation value of the attention pixel to further circuitry for usein generating a display image that corresponds to a sensed image thatwas sensed by the image sensor.
 2. The image processing apparatusaccording to claim 1, wherein the circuitry is further configured todetermine whether or not one or more saturation pixel values are presentin the pixel values of the interpolation W pixels, wherein if thesaturation pixel value is not present in the pixel values of theinterpolation W pixels, the circuitry calculates a compensation pixelvalue of the attention pixel by executing the smoothing processing towhich the smoothing weight, calculated based on the pixel values of theinterpolation W pixels, is applied, and wherein if the saturation pixelvalue is present in the pixel values of the interpolation W pixels, thecircuitry calculates the compensation pixel value of the attention pixelby executing the smoothing processing to which the smoothing weight,calculated based on each pixel value of the attention pixel that is thecompensation target, and of the reference pixel which has the same coloras the attention pixel within the reference area, is applied withoutapplying the interpolation W pixels.
 3. The image processing apparatusaccording to claim 1, wherein the circuitry is further configured toexecute processing that interpolates the W pixel at the position of thereference pixel which has the same color as the attention pixel presentin the reference area that is a two-dimensional area with the referencearea as the two-dimensional area.
 4. The image processing apparatusaccording to claim 1, wherein the circuitry is further configured toexecute processing that interpolates the W pixel at the position of thereference pixel which has the same color as the attention pixel presentin the reference area that is a one-dimensional area with the referencearea as the one-dimensional area.
 5. The image processing apparatusaccording to claim 1, wherein the circuitry is further configured toexecute pixel value compensation as noise reduction (NR) processing thatreduces noise that is included in the attention pixel.
 6. The imageprocessing apparatus according to claim 1, wherein the circuitry isfurther configured to determine a likelihood of a defect, that is,determines whether or not there is a likelihood that the color pixelwill be a defective pixel, wherein the circuitry is further configuredto execute texture detection processing that determines whether or notthe color pixel is in a texture area, by applying a W pixel in theneighborhood of the color pixel that is determined as having thelikelihood of the defect, wherein in the texture detection processing,if it is determined that the color pixel is in the texture area, thecircuitry does not execute defect compensation processing, and whereinin the texture detection processing, if it is determined that the colorpixel is not in the texture area, the circuitry executes the defectcompensation processing.
 7. The image processing apparatus according toclaim 6, wherein in the texture detection processing, the circuitrydetermines whether or not the color pixel is in the texture area, byapplying a difference in the pixel value between a W pixel that isclosest to the color pixel that is determined as having the likelihoodof the defect, and a W pixel outside of the closest W pixel.
 8. An imageprocessing method of executing pixel value compensation in an imageprocessing device, wherein the image processing device performs:receiving image data acquired by an image sensor in an RGBW arrangementthat has each color pixel of R, G, and B and a white (W) pixel thatpasses through almost all wavelength light of each wavelength of the R,G, and B; interpolating W pixels at a position of an attention pixelthat is a compensation target, and at a position of a reference pixelwhich has the same color as the attention pixel within a reference area,in a pixel value compensation processing of a color pixel; calculatingsmoothing weight based on pixel values of the interpolation W pixels;calculating a compensation pixel value of the attention pixel byexecuting smoothing processing to which the calculated smoothing weightis applied, wherein the smoothing processing comprises computing aproduct of (a) a pixel value of the reference pixel which has the samecolor as the attention pixel and (b) a difference between the pixelvalue of the interpolation W pixel at the position of the attentionpixel and the pixel value of the interpolation W pixel at the positionat the reference pixel; and providing the compensation value of theattention pixel to circuitry for use in generating a display image thatcorrespond to a sensed image that was sensed by the image sensor.
 9. Anon-transitory computer readable medium encoded with instructions which,when executed by a processor of an image processing apparatus, cause theimage processing apparatus to execute pixel value compensationcomprising acts of: receiving image data acquired by an image sensor inan RGBW arrangement that has each color pixel of R, G, and B and a white(W) pixel that passes through almost all wavelength light of eachwavelength of the R, G, and B; interpolating W pixels at a position ofan attention pixel that is a compensation target, and at a position of areference pixel which has the same color as the attention pixel within areference area, in a pixel value compensation processing of a colorpixel; calculating smoothing weight based on pixel values of theinterpolation W pixels; calculating a compensation pixel value of theattention pixel by executing smoothing processing to which thecalculated smoothing weight is applied, wherein the smoothing processingcomprises computing a product of (a) a pixel value of the referencepixel which has the same color as the attention pixel and (b) adifference between the pixel value of the interpolation W pixel at theposition of the attention pixel and the pixel value of the interpolationW pixel at the position of the reference pixel; and providing thecompensation value of the attention pixel to circuitry for use ingenerating a display image that corresponds to a sensed image that wassensed by the image sensor.
 10. The image processing apparatus accordingto claim 1, wherein the circuitry comprises a programmed computer.